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Issue 1: The State Misspent Federal Funds On Oversized Routers Which 
Could Have Been Used To Expand The State's Broadband Fiber Network. 

The Office of Technology (OT) and the Broadband Technology Opportunity Program 
(BTOP) Grant Implementation Team (GIT) made the decision to purchase 1,164 Cisco model 
3945 routers for many Community Anchor Institutions (CAIs) when smaller routers would have 
been appropriate for hundreds of the CAIs. The GIT was assembled in 2009 to apply for and 
then administer the federal grant awarded through the federal government's Broadband 
Technology Opportunity Program. However, according to Cisco's own literature, the model 
3945 router was not an appropriate choice for the majority of the 1,164 CAIs that have or are to 
receive the new routers. The GIT's decision to spend the federal funds on oversized routers 
resulted in millions of dollars in federal funds not being spent on expanding the state's fiber optic 
broadband network. 

Routers are generally classified by levels. Level 1 routers are typically used in single 
family homes and small businesses, while Level 5 routers are used at the core of the internet. 
The Cisco model 3945 routers are Level 3 routers. Level 1 routers would have enabled most of 
the state's libraries to provide quality broadband service to their communities. Level 2 routers 
are presently allowing State Police detachments to effectively communicate across the State 
Police's computer system. Level 2 routers would have allowed at least 368 West Virginia 
schools, which received the Cisco model 3945 routers, to meet the 2017 standards for broadband 
access set by the national State Educational Technology Directors Association. 1 

A capacity and a users need survey, prior to the procurement of the routers, would have 
determined the appropriate router size for the CAIs; but, such surveys were not conducted by the 
GIT or the OT. Since the state's libraries and schools received the majority of the routers, the 
Legislative Auditor's Office focused its analysis on these entities. However, the Legislative 
Auditor's Office also reviewed the needs of the State Police, the Supreme Court of Appeals, and 

'Broadband Imperative - Recommendations to Address K-12 Education Infrastructure Needs - State Educational 
Technology Directors Association. 
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the state's 55 county governments. This included the Legislative Auditor working with the West 
Virginia Association of Counties to survey the 55 county governments and with the State 
Department of Education to survey the 55 state school districts as to their use of VoIP phone 
systems. Based on the Legislative Auditor's research, some conclusions can be readily drawn: 

1) Smaller less expensive routers could have been purchased for the state's 172 
libraries. If the average cost savings was $16,265 less per router, $2.8 million 
could have been saved. 

2) Smaller less expensive routers, if even necessary, could have been purchased 
for the State Police for $15,000 less per router saving $1 million. 
Furthermore, if the GIT and OT had not purchased 70 of the Cisco model 
3945 routers, the State would have saved approximately $1.4 million. 



3) Several of the State's public schools are presently able to meet the 2017 Broadband 
standards set by the national State Educational Technology Directors Association. In 
the opinion of the Legislative Auditor, routers significantly smaller than the Cisco 
model 3945 could have been used to ensure almost all of the State's schools meet 
these standards. Purchasing appropriately sized routers, which could have cost 
$10,000 less, for at least the 368 schools with an enrollment less than 500 which 
received Cisco model 3945 routers, could have achieved the same result for $3.68 
million less. 

Voice Over Internet Protocol (VoIP) 

The Office of Technology asserts that the opportunity the routers provide for CAIs to 
save on phone bills supports the appropriateness of the router purchase. According to the State's 
present Chief Technology Officer, Gale Given: 
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Studies have shown that VoIP as compared to Public Switched Telephone Network 
service can save up to 40% on local calls and up to 90% on long distance calls. 2 

Voice Over Internet Protocol, VoIP, is a method to make phone calls using the internet. 
VoIP provides the method for converting analog voice communications to digital form so that it 
can be transmitted across the internet. VoIP enables anyone with a reasonable quality internet 
connection to receive phone service delivered through their internet connection instead of from 
the local phone company. The main benefit of VoIP is cost savings. Companies such as Vonage 
and Suddenlink offer consumers VoIP plans. The Legislature uses a VoIP phone system 
throughout its offices. 

The Cisco model 3945 routers are capable of handling VoIP phone systems. In fact, 
according to Mark Williamson of Cisco, "VoIP is just one of the many services that the 3945 can 
support." 

In examining whether the Cisco model 3945 routers should have been purchased, the size 
of the VoIP phone system to be supported should have been one consideration. Cisco's 
Telephony Using Cisco U CM Deployment Guide (August 2012) states: 

Because Cisco Integrated Services Routers Generation 2 (ISR G2) have different 
processing capabilities based on the number of phones and features applied, it is 
important to select the appropriate platform based upon expected usage.... [Emphasis 
added.] 

Table 1 shows the recommended Cisco router based upon the number of voice gateways for a 
VoIP system. The table also includes the list price according to a Cisco representative for the 
router on February 3, 2013, for comparison purposes. Cisco recommends its model 3945 routers 
for remote sites with 700 to 1,200 phones. Thus, the 1,164 Cisco model 3945 routers which the 
State purchased can support between 814,000 and 1.39 million VoIP phones. This greatly 
exceeds the needs of the State and its political subdivisions for which the routers were purchased. 



2 Given 1-24-2013 Letter From In Response to 1-17-2013 letter.pdf 
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Cisco Recommendations 
For Appropriate Router For Voice Gateways 




Number of Phones 


Recommended Cisco Router 


List Price 4 


1 


-4 


881-V-K9 


$ 1,795 


5- 


50 


2911-VSEC/K9 


$ 4,395 


50- 


100 


2921-VSEC/K9 


$ 5,495 


100 


-250 


2951-VSEC/K9 


$ 9,395 


250- 


-700 


3925-VSEC/K9 


$ 11,995 


700- 


1200 


3945-VSEC/K9 


$ 15,495 


Source: Cisco 


information see Footnotes 3 and 4 below. 







Virtual Private Networks (VPN) 

Cisco also recommends routers based on the number of users with regards to VPN 
tunnels. The Legislative Auditor examined Cisco's own recommendations in two of its 
publications: Cisco Router Guide: For teleworkers, small offices, small to medium-sized 
businesses, and enterprise branch and head offices (Summer 2010 V.6) and Cisco Integrated 
Services Routers - Performance Overview (White Paper 2010). Both Cisco documents address a 
Cisco Web VPN feature, called SSL VPN, which allows a user to securely access resources on a 
corporate local area network (LAN) from anywhere with an SSL-enabled web browser. VPN is 
an acronym for virtual private network. Because VPN is encrypted, a router has to unencrypt the 
data that comes across the VPN connection or tunnel. This process uses more of a router's 
resources than unencrypted communications. Therefore, it is the opinion of the Legislative 
Auditor that a router which can simultaneously support 200 VPN users could easily support the 
same number of internal users. Table 2 indicates the number of VPN users each model of Cisco 
router is designed to support, which is a fair proxy for the number of internal users each router 
could support. 



3 Cisco's Telephony Using Cisco UCM Deployment Guide (August 2012), page 4. 

4 Cisco Systems Inc. list prices according to Mark Williamson, of Cisco, February 3, 2013 email to the Legislative 
Auditor. 
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laDie z 

Simultaneous VPN Tunnel Users Supported By Cisco's Routers 


Router Model 


Simultaneous Users 


880 


10 


890 


25 


1921 


50 


1941 


75 


2901 


75 


2911 


100 


2921 


100 


2951 


150 


3925 


200 


3925E 


500 


3945 


200 


3945E 


500 


Source: Cisco Router Guide: For teleworkers, small offices, small to medium-sized 
businesses, and enterprise branch and head offices (Summer 2010 V.6) and Cisco 
Integrated Services Routers - Performance Overview (White Paper 2010). 



According to one of the Legislature's network specialists, in designing either a Wide 
Area Network (WAN) or a Local Area Network (LAN) there has to be great consideration given 
to usage, bandwidth, and the critical nature of the data the network must accommodate. There 
must always be some extra capacity built into every aspect of every network but no network 
specialist would recommend network capacity to handle 100% of network maximum at all times. 
All users would never be accessing the network at 100% capacity simultaneously. In the normal 
course of daily business there are always distractions that take people away from the actual 
network from time to time. No user is downloading or uploading at capacity more than a few 
moments at a time. There should never be a time when 100% of the network users needed 100% 
of the rated bandwidth available for any extended period of time. The bottleneck or slowdown of 
the work-flow in such an extreme scenario would be the work capacity of the human beings 
themselves, not the network. Even in a 'mission critical' network, designing and engineering for 
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100% capacity to be in use 24/7 would be cost prohibitive to most if not all businesses. 
Furthermore, Cisco's own web site clearly states that the Cisco 3900 Series ISRs are: 

Ideal for mid-size to large deployments 5 
and that the 

Cisco 870 Series routers are ideal for small offices, up to 20 users. 6 

In fact, Prashanth Shenoy, the Senior Marketing Manager at Cisco stated in his corporate blog 
that a Cisco model 3945 router can support a 150 person branch office. The Legislative 
Auditor finds that libraries, State Police detachments and many schools do not meet Cisco's own 
criteria for mid-sized or large deployments. 

Public Libraries 

GIT assigned 172 public libraries in the State of West Virginia a Level 3 model 3945 
Cisco branch router through the BTOP program. The Library Commission provided the 
Legislative Auditor with information on the number of phone lines at 105 of the state's public 
libraries. Ninety-one (91) of the libraries had one to three phone lines. The largest library, for 
which the Library Commission provided data, the Cabell County Library, had only 15 phone 
lines. The data with regards to public libraries' phones are shown in Table 3. 



5 Unified Communications on Cisco Integrated Services Routers, on Cisco website, 
http://www.cisco.com/en/US/prod/voicesw/ps6790/uc isr.html . 
6 Cisco 800 Series Integrated Service Router, on Cisco website, 

http://www.cisco.com/en/US/prod/collateral/routers/ps380/ps6200/prod qas0900aecd8028a982 ps380 Products Q 
and A Item.html 

7 Prashanth Shenoy, Why Cisco, Not Juniper?, http://blogs.cisco.com/author/PrashanthShenoy/ 
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# Phone Lines 


# Libraries 


% 


Cumulative % 


1 


51 


52.0 


52.0 


2 


33 


33.7 


85.7 


3 


7 


7.1 


92.9 


4 


2 


2.0 


94.9 


5 


1 


1.0 


95.9 


6 


2 


2.0 


98.00 


8 


1 


1.0 


99.00 


15 


1 


1.0 


100.00 


Total 


105 


100% 




Source: West Virginia Library Commission as of February 7, 2013 - (unaudited) 



The Library Commission also provided the Legislative Auditor with information on the number 
of internet user connections at each of the state's 172 public libraries. This data is shown in 
Table 4. 







Table 4 








er West Virginia Public Library 


User 












Connections 


# of Libraries 






% 


Cumulative % 


1-5 


26 


15.1 


15.1 


6- 10 


73 


42.4 


57.5 


11-15 


33 


19.2 


76.7 


16-20 


18 


10.5 


87.2 


21-25 


8 


4.7 


91.8 


26-30 


2 


1.2 


93.0 


31-35 


3 


1.7 


94.8 


36-40 


1 


0.6 


95.3 


41-45 





0.0 


95.3 


46-50 


3 


1.7 


97.1 


50 + 


5 


2.9 


100 


Total 


172 


100 




Source: West Virginia 


Library Commission 


(unaudited) 
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As shown in Tables 3 and 4, 87% of the public libraries have 20 or fewer phone lines and 
57% of the public libraries have 10 or fewer internet connections. According to Cisco's own 
literature, a Cisco 291 1 router could easily support the future internet and VoIP needs for each of 
the 172 libraries at a cost of approximately $16,265 per router less than the Cisco model 3945 
router which the State purchased. If the GIT and OT had purchased Cisco model 2911 routers 
for state's public libraries, the state could have saved $2.8 million. 

Education 8 

According to documentation provided by the Department of Education, 338 schools 
currently do not meet the 2014 or 2017 recommended standards for broadband as set forth 
by the national State Educational Technology Directors Association. The 2014 standard is 
connection speeds of at least 100 Mbps per 1,000 students and staff, and the 2017 standard is 
connection speeds of at least 1 Gbps per 1,000 students and staff. Reasons schools may fail to 
meet the standards despite the technology investment include: failure to connect the fiber to the 
router; an unexpired contract with a current service provider; or an inability on the part of the 
local school board to pay higher fees for the service contract. 

Of the schools that do not meet the standards for either year, 102 (30%) have fiber 
completed and a Cisco model 3945 router installed from the BTOP grant. All schools that 
received BTOP fiber and routers that meet the 2017 standard have student populations of 100 or 
fewer students. Because the standards are based upon students to broadband speed ratio, a lower 
enrollment makes meeting the standards easier. 

However, 11 schools with more than 100 students meet the 2017 standards without fiber 
or routers from the BTOP grant. These schools all had preexisting fiber lines, but pay higher 
prices for higher internet speeds than the schools who do not meet the standards. The key factor 
is the speed of its internet service not the size of its router. 



School data was obtained from the West Virginia Department of Education and includes career centers, technical 
centers, alternative schools, and vocational center. 
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In examining the appropriateness of purchasing the Cisco model 3945 routers for 457 of 
the state's public schools, the Legislative Auditor examined whether the schools could meet the 
2017 standards with a smaller Cisco router. For 368 of these schools, with an enrollment of 499 
or less, it is the opinion of the Legislative Auditor that these schools could meet the 2017 
standards with a Cisco 2951 router. As for the 89 schools which received or are scheduled to 
receive a Cisco model 3945 router with an enrollment of 500 or more students, the Legislative 
Auditor does not question the purchase of the Cisco model 3945 routers. 9 However, if the GIT 
and OT had purchased 368 Cisco model 2951 routers, the State would have saved approximately 
$10,000 per router or approximately $3.68 million. 



State Police 



The GIT assigned 77 of the Cisco model 3945 routers to the West Virginia State Police 
(WVSP). On its January 31, 2013 payroll, the WVSP had 1,057 total employees, 662 of whom 
were sworn law enforcement officers. According to Bill Gallagher, Director of Information 
Technology for the WVSP, the WVSP has approximately 1,500 internet connections throughout 
the state. Given the size of the WVSP, the Legislative Auditor finds that the Cisco model 3945 
routers were significantly oversized for the WVSP's needs. The WVSP has 67 detachments and 
was assigned 77 Cisco model 3945 routers. On average, one Cisco model 3945 router was 
purchased for every 13.7 WVSP employees and every 19.5 internet connections. Yet, as 
referenced earlier, the Cisco model 3945 is "ideal for mid-size to large deployments" and can 
support 200-plus simultaneous computer connections. The number of employees at WVSP 
detachments is shown in Table 5. 



9 The Legislative Auditor notes that a capacity and users need survey should have been conducted to determine if 
even schools with enrollment over 500 require a Cisco model 3945 router. 
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Table 5 

Staffing Levels at WVSP Detachments 


Employees At Detachment 


Number of Detachments 


1-5 


17 


6-10 


28 


11-15 


10 


16-20 


4 


Over 20 


8 


Total 


67 


Source: West Virginia State Police (Unaudited) 



According to Cisco's own literature, Cisco model 2921 would be a more appropriate 
device for the State Police detachments than the Cisco model 3945 routers which the State 
purchased at a cost of $20,661 each. The Cisco model 2921 's list price is $5,495. Furthermore, 
according to Mr. Gallagher of the WVSP: 

1) The WVSP conducted a study and purchased appropriate routers 4 years ago. The 
majority of routers purchased were Cisco model 2801s, with Cisco model 2821s for the 
medium sized detachments, and one Cisco model 3845 router. 

2) The WVSP was never contacted by the BTOP Grant Implementation Team concerning 
the WVSP's broadband and telephone needs. 

3) The WVSP did not ask for the Cisco model 3945 routers. Instead the WVSP were simply 
informed that the WVSP was receiving 77 routers. 

Mr. Gallagher also informed the Legislative Auditor: 

We have deployed all of the BTOP routers as per the NTIA definition of 
deployed. This definition supplied to the State Police is essentially, the router is 
on site and received by the anchor tenant. However only 2 of the routers 
assigned to the WV State Police are installed and working. This is due to 
telephony needs that are not met by the 3945 routers as supplied to the WV State 
Police. The two which are installed and working were upgraded and paid for by 
the State Police to meet our telephony needs, (emphasis added.) 
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The WVSP has yet to be able to use the Cisco model 3945 routers for their VoIP system, 
because the Cisco model 3945 routers purchased, did not include the appropriate Cisco VoIP 
modules. Due to this fact, the Office of Technology has a low bid of $84,768 to purchase the 
necessary additional equipment for the WVSP Cisco model 3945 routers. 

It is the opinion of the Legislative Auditor that the WVSP did not need the 77 routers, 
given that they had conducted a study, purchased appropriate routers year 4 years ago, and are 
presently functioning without using 75 of the Cisco model 3945 routers. If the GIT and OT had 
not purchased 70 of the Cisco model 3945 routers, the State would have saved approximately 
$1.4 million. 

Examples of Questionable Placement of Cisco Model 3945 Routers 

The Legislative Auditor found numerous examples of Cisco model 3945 routers being 
placed, not only in small facilities, but also multiple routers were concentrated in areas with 
small populations. The following are two examples of high concentrations of BTOP routers in 
communities with small populations. 
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The city of Clay in Clay County received 7 total routers to serve a population of 491. 
Five of these routers are located within .44 miles of the each other. 




1U 2010 Census Data. 
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The city of Franklin in Pendleton County received 6 total routers to serve a population of 
721. 11 Four of these routers are located within .76 miles of each other. 




11 2010 Census Data. 
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In addition, to the high concentration of routers in small towns, the Legislative Auditor also 
questions the chosen locations for the routers. For example, one of the Cisco model 3945 routers 
was placed in the Marmet Public Library. The Marmet Public Library is an extremely small 
facility with only one internet connection. 




However, Riverside High School which serves the Marmet area and has an enrollment of 
1,244 students and multiple internet connections did not receive a router. This problem is 
statewide. The Legislative Auditor found 36 out of the 57 state schools with enrollment over 
750 students (63%), which could most appropriately use the Cisco model 3945 routers, are not 
being provided a router. Table 6 shows the 36 state schools with an enrollment of 750 students 
which did not receive a Cisco model 3945 router. 
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Table 6 

Schools With Over 750 Students Which Did Not Receive A Cisco Model 3945 Router 




County 


School Name 


Enrollment 


Current Bandwidth Mbps 


Meets 2014 
Standard 


Meets 2017 
Standard 


Cabell 


CABELL MIDLAND HS 


1836 


100 


NO 


NO 


Wood 


PARKERSBURG HS 


1783 


100 




NO 


Berkeley 


HEDGESVILLE HS 


1691 


200 


YES 


NO 


Berkeley 


MARTINSBURG HS 


1685 


200 


YES 


NO 


Monongalia 


MORGANTOWN HS 


1616 


100 


NO 


NO 


Wood 


PARKERSBURG SOUTH HS 


1580 


100 


NO 


NO 


Berkeley 


MUSSELMAN HS 


1557 


200 


YES 


NO 


Cabell 


HUNTINGTON HS 


1556 


100 


NO 


NO 


Jefferson 


JEFFERSON HS 


1367 


100 


NO 


NO 


Monongalia 


UNIVERSITY HS 


1281 


100 


NO 


NO 


Kanawha 


RIVERSIDE HS 


1244 


50 


NO 


NO 


Kanawha 


CAPITAL HS 


1236 


50 


NO 


NO 


Mason 


POINT PLEASANT JR/SR HS 


1214 


100 


NO 


NO 


Berkeley 


MUSSELMAN MIDDLE 


1157 


200 


YES 


NO 


Putnam 


HURRICANE HS 


1142 


20 


NO 


NO 


Greenbrier 


GREENBRIER EAST HS 


1129 


100 


NO 


NO 


Kanawha 


GEORGE WASHINGTON HS 


1115 


50 


NO 


NO 


Jefferson 


WASHINGTON HS 


1110 


200 


YES 


NO 


Mercer 


PRINCETON HS 


1075 


100 


NO 


NO 


Kanawha 


SAINT ALBANS HS 


1070 


50 


NO 


NO 


Kanawha 


SOUTH CHARLESTON HS 


1036 


50 


NO 


NO 


Jackson 


RIPLEY HIGH SCHOOL 


962 


100 


YES 


NO 


Berkeley 


MARTINSBURG S. MIDDLE 


875 


200 


YES 


NO 


Greenbrier 


EAST.GREENBRIER MIDDLE 


865 


100 


YES 


NO 


Putnam 


HURRICANE MIDDLE 


865 


20 




NO 


Berkeley 


BERKELEY HEIGHTS ELEM. 


846 


200 


YES NO 


Logan 


LOGAN HS 


826 


100 


YES 


NO 


Putnam 


WINFJELD HS 


823 


20 


NO 


NO 


Monongalia 


CHEAT LAKE ELEM. 


812 


10 


NO 


NO 


Harrison 


BRIDGEPORT HS 


794 


1000 


YES 


YES 


Nicholas 


NICHOLAS HS 


792 


45 


NO 


NO 


Jackson 


RIPLEY MIDDLE 


776 


100 


YES 


NO 


Harrison 


ROBERT C. BYRD HS 


775 


1000 


YES 


YES 


Cabell 


BARBOURSVILLE MIDDLE 


771 


100 


YES 


NO 


Monongalia 


MOUNTAINVIEW ELEM. 


760 


10 


NO 


NO 


Kanawha 


NITRO HS 


755 


50 


NO 


NO 


Source: West Virginia State Department of Education (Unaudited) 
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Cause 

In attempting to determine the cause of the purchase of oversized routers for hundreds of 
facilities, the Legislative Auditor held discussions and corresponded with multiple State agencies 
and individuals concerning their involvement in the project. These include: members of the 
original GIT still employed by the State, Mr. Jimmy Gianato, Director of Homeland Security and 
Emergency Management Division and Lt. Colonel Michael Todorovich; the Cisco engineer for 
the project, Mr. Mark Williamson; Mr. David Tincher, Director of the State Purchasing Division; 
the present State Chief Technology Officer, Ms. Gale Given; John Dunlap with the Office of 
Technology; the State Department of Education; the State Supreme Court of Appeals; the State 
Library Commission; and, the State's former Chief Technology Officer at the time of the 
purchase of the routers, Mr. Kyle Schafer. 

According to Mr. Gianato and Mr. Williamson, Cisco recommended the State purchase 
the Cisco 3945 routers because the DOE required that the routers have internal dual power 
supplies. Mr. Williamson informed the Legislative Auditor that: 

The 3 Agencies we consulted with gave us their requirements, and we proposed a branch 
router platform to meet the requirements that were conveyed to us. The State ultimately 
selected that platform. 

The Legislative Auditor then asked Mr. Williamson specifically who requested dual power 
supplies. Mr. Williamson stated: 

That was the Dept. of Education initially (Greg Chapman and Eric Petrucci), and agreed 
upon by the Office of Technology (John Dunlap), and then HLS (Jimmy Gianato). 

However, Mr. Williamson was unable to provide the Legislative Auditor with any emails or 
other documentation from the Department of Education employees showing that they asked for 
dual power supplies in the routers. Thus, the Legislative Auditor contacted DOE, OT, and GIT 
officials, including Mr. Chapman, Mr. Petrucci, Mr. Dunlap, and Mr. Gianato. 
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First, Ms. Brenda Williams, Executive Director of the Office of Instructional Technology 
and Mr. John Miller, Assistant Director of the Office of Instructional Technology representatives 
from the Department of Education informed the Legislative Auditor: 



The Department of Education did not request or require that the routers for the state 's 
schools have internal dual power supplies. Education would not have made this 
requirement because unless a school has two power sources the feature of dual power 
supplies would have no use. The BTOP grant implementation team did not ask 
Education to provide information on which schools had dual power supplies. 

Mr. Petrucci, a network engineer for the DOE, informed the Legislative Auditor: 

/ did not request or require that BTOP purchased routers for the schools have internal 
dual power supplies. We have never recommended that schools use routers with dual 
power supplies. 

Mr. Chapman, another network engineer for the DOE, informed the Legislative Auditor that to 
"the best of my knowledge" he did not discuss the BTOP purchased routers with dual power 
supplies with the BTOP Grant Implementation team. 

The Legislative Auditor also contacted Mr. Dunlap, with the State Office of Technology. 
Mr. Dunlap stated: 

It is my understanding meetings were held with various state entities including the Office 
of Technology to discuss router features with (Cisco representatives) Mr. Dailey and Mr. 
Williamson. Once these discussions were completed, they recommended the 3945 router 
Build of Material (BoM) to the state. 

Features such as dual power supplies for 24/7/365 locations such as regional jails and 
DHHR state hospitals were discussed and requested. Also, discussions of possible VOIP 
deployments for other OT supported locations occurred but not required. It was never 
implied to put each feature in all routers for WVOT anchor institutions. 



19 lp g 



West Virginia Legislative Auditor Performance Evaluation & Research Division 



Mr. Williamson (of Cisco) also informed the Legislative Auditor that: 

The meetings with Dept. of Education on June 14, 2010 and with Office of Technology on 
June 15, 2010 resulted in a series of Spreadsheets that were created, and reviewed by the 
State. The dual power supplies, along with the NAM, EtherSwitch, etc. were listed on 
every revision of those Spreadsheets that were submitted for review. This review process 
was done to insure that the configuration shown met their requirements. The first of 
those spreadsheets, created on June 16 2010 is attached. We always conduct this type of 
review when working with our Customers and Partners, to confirm that they are ordering 
exactly what they want. 

The Legislative Auditor subsequently emailed Mr. Williamson tables showing Cisco's own 
standards for their routers and asked: 

Given Cisco's own statement that 'it is important to select the appropriate platform 
based upon expected usage' and Cisco's own recommended standards, I would 
appreciate an explanation as to why you believe the 3945 routers are not oversized and 
misconfigured for hundreds of locations; and, thus, a significant over expenditure of 
millions of dollars for Cisco equipment. 

Mr. Williamson (of Cisco) responded: 

We based our initial recommendation on the requirements that were conveyed to us. 
Those requirements were Dual Power Supply, Network Analysis Module, 16-port 
EtherSwitch Module, 2-port Tl VW1C, and DSP's for Voice Trunk support. Those 
requirements were conveyed to us at the meetings on June 14, 2010 and June 15, 2010. 
Those requirements were captured on the initial spreadsheets, and submitted to the State 
for review beginning on June 17, 2010. Based on those requirements, the Branch router 
that can accommodate the two Service Modules and Dual Power are the 3925 and 3945. 
The 3925 supports two Service Modules, and therefore would have been full, so the 3945 
was suggested in order to accommodate additional Service Modules in the future. The 
State was asked to offer any alternative configuration they saw fit. This review process 
was done to insure that the configuration shown accurately captured the requirements 
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that were discussed, and met the State 's requirements. We were prepared to show any 
alternative configurations. 



The Legislative Auditor also discussed the router purchase with Mr. Kyle Schafer, the 
State's former Chief Technology Officer. According to Mr. Schafer: 

OT did not participate in any of the meetings that lead up to this decision. The first I saw 
of this purchase order was when it was sent to OT for approval. That 's when I sent it 
back to John Dunlapfor additional review. That's also when I went to (former Secretary 
of the Department of Administration) Ferguson with our concerns that many of these 
devices may be oversized based on their designated locations. 

In fact, Mr. Dunlap expressed OT's concerns on July 12, 2010 in an email to Mr. Gianato 
and Colonel Todorovich with the routers proposed. Mr. Dunlap's email states: 

Since a site assessment has not been conducted for the 1064 locations, the Office of 
Technology is concerned that this equipment may be grossly oversized for several of the 
facilities in which it is currently slated to be installed. As a result, the Office of 
Technology would like to evaluate these facilities and make recommendations to deploy 
the 3900 series routers where it may be better utilized .... 

Mr. Schafer then met with Kelly Goes, Secretary of Commerce, on July 13, 2010. On 
July 13, 2010, Mr. Dunlap emailed Mr. Todorovich that "The Office of Technology will proceed 
with the purchase order to acquire 1064 routers..." However, Mr. Schafer's formal approval to 
purchase the routers is dated July 6, 2010. The Legislative Auditor notes that Mr. Schafer 
approved the purchase as the State's Chief Technology Officer under the authority of §5A-6- 
4(a)(3) of the West Virginia Code. 

While Mr. Williamson of Cisco and Mr. Gianato of the GIT stated a need for the routers 
to have a dual power supply, the ultimate cause of the state purchasing inappropriately sized 
routers is that neither a capacity study nor a user need study was conducted. A capacity study of 
the statewide wide area network (WAN) would have identified: 1) existing network capabilities; 
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2) future network capacity needs, and 3) the parties responsible for each segment of the statewide 
WAN. A users need study would have attempted to determine present and future needs and 
desires from broadband for: 

1) Voice over Internet Protocol, including the need for: 4-digit dialing, eliminating 
dependency on and payments to providers of long distance services; 

2) Video streaming; 

3) Virtual meetings; 

4) Virtual class rooms; 

5) Entertainment; 

6) Mass data uploads/downloads; 

7) Increased hard- wired internet connections; and, 

8) Creation of wireless community broadband services. 

Unfortunately, no studies were ever conducted. Furthermore, according to the State 
Supreme Court of Appeals and the State Police, no one from the BTOP grant implementation 
team ever contacted them until after the routers were purchased. Furthermore, according to the 
Library Commission, there were no written communications with GIT or OT regarding a need 
for routers. 

Lt. Colonel Todorovich, a member of the GIT, informed the Legislative Auditor 
that he agreed with the statement: 

Those making the decisions on how to spend the money did not consult individuals 
with technical knowledge on the best methods to utilize the funds. 

Finally, inappropriately sized routers were purchased because the State Purchasing 
Division allowed the Office of Technology to use a purchasing process unauthorized by either 
statute or legislative rule. This purchasing process, as discussed in Issue 2, prevented other 
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vendors such as HP, Brocade, Juniper, and Alcatel-Lucent from bidding on this statewide 
project. 



The Grant Implementation Team Purchased Over $6.6 million Worth of 
Additional Features for Each Cisco Model 3945 Router 



In addition to purchasing routers that are deemed to be oversized for many locations, the 
BTOP Grant Implementation Team also purchased $6,667,215 worth of additional features for 
each of the 1,164 Cisco model 3945 routers. This was an additional cost of $5,728 per router. 
Below is a brief summary of each feature. Table 7 displays the total cost for each feature. 



Table 7 


Additional Cisco Model 3945 Router Features 


Part 


Amount 


Data Paper Pak 


$540,512 


Tl Card 


$1,081,024 


ENCHD EtherS witch 


$1,431,814 


IP Services License Upgrade 


$1,438,305 


Power Supply with power over 


$270,256 


Power Supply with POE 


$540,512 


Console cable - USB 


$16,215 


Voice Security Bundle 


$1,348,577 


Total 


$6,667,215 


Source: Verizon bid 



The Legislative Auditor questions the need for each of these items in each Cisco model 
3945 router configuration. When asked for the justification for purchasing each of the options 
Chief Technology Officer Gale Given stated: 

The team decided to have all routers identically equipped. 

As stated previously, the GIT did not conduct a user needs survey to determine whether 
the options were necessary currently or whether there would be a future need. Notably, the GIT 
purchased a Cisco voice security bundle option for each router at a total cost of $1,348,577. The 
voice security bundle option would enable the router to utilize a Cisco VoIP solution if the 
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Community Anchor Institution purchased a VoIP system. Chief Technology Officer Gale Given 
justified the inclusion of the voice security bundle option by stating: 

Studies have shown that using VoIP as compared to Public Switched Telephone Network 
Service can save up to 40% on local calls and up to 90% on long distance calls. 

The Legislative Auditor agrees that in some cases, entities can save money on telephone 
service by switching to VoIP. However, it is unknown at this time how many of the router 
recipients will actually upgrade to a VoIP system. Thus, this appears to be a questionable 
expenditure of over $1.3 million. As a result of the GIT not conducting its own users needs 
survey, the Legislative Auditor conducted a survey of the 55 County Clerks and the technology 
officers in each of the 55 county Board of Education offices regarding VoIP. Responses were 
received from 38 of the County Clerks and 32 of the county school technology officers. Seven 
of the County Clerks stated that their offices currently have VoIP, while 12 of the county school 
systems currently have VoIP. As far as future plans, 4 County Clerks responded that there are 
plans for VoIP in the next 5 years, while 6 county schools systems have future plans for VoIP. 
Thus, based on the counties that responded to the survey, a minimum of 25 County Clerks and 
11 county school systems have no plans to have VoIP in the next 5 years. One County Clerk 
responded that they did not even know what VoIP is. Survey results are displayed in Table 8. 
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Table 8 

Results of the Legislative Auditor's User Needs Survey for VoIP 



County Clerks Offices 



County School Systems 



VoIP currently 



Future plans 



VoIP currently 



Future plans 



Cabell 



Barbour 



Braxton 



Boone 



Harrison 



Hancock 



Clay 



Brooke 



Jackson 



Hardy 



Doddridge 



Kanawha 



Kanawha 



Jefferson 



Gilmer 



Lewis 



Mercer 



Hancock 



Logan 



Mineral 



Harrison 



Wirt 



Randolph 



Lincoln 



Mingo 



Monongalia 



Pendleton 



Pleasants 



Upshur 



Source: Summary of survey conducted by the Legislative Auditor 's Office (unaudited) 



According to a representative of the West Virginia Library Commission, there are 
currently no public libraries on the State library network that utilize VoIP. He further stated that 
the VoIP modules have no current value if the library does not have or will not have VoIP. The 
Legislative Auditor finds that many libraries will not have a need to utilize VoIP since many of 
the libraries are relatively small with few employees. 

Effect 

The effect of purchasing unnecessarily large routers is two fold. First, funds spent on 
unnecessary hardware, are funds which could have been used to build more fiber throughout 
West Virginia. If the BTOP implementation team overspent at least $5 million on routers, with 
an average cost of $47,860 for every mile of fiber laid, this $5 million could have resulted in 104 
additional miles of broadband fiber in the State of West Virginia. In fact, the State's current 
Chief Technology Officer, Ms. Gale Given, agrees with the Legislative Auditor's opinion that: 

BTOP funds could have been diverted from routers to middle mile and/or other 
broadband initiatives with NTIA approval. 
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Second, the cost of maintaining the Cisco 3945 routers after the extended maintenance 
expires may exceed the cost of simply buying new appropriately sized routers. As discussed 
previously, the cost to retrofit the State Police Cisco 3945 routers with the appropriate Cisco 
modules to allow the model 3945s to work with the State Police VOIP system is approximately 
the same cost as simply buying new small routers. 

Conclusion 

The Legislative Auditor found that some State agencies did conduct analysis to determine 
router sizes prior to making purchases. For example, the State Tax Department purchased 55 
Cisco model 2821 routers for installation in the 55 county courthouses. The Tax Department 
consulted with other State agencies as well as Verizon prior to the purchase of Cisco model 2821 
routers in 2008, using the WAN08 statewide contract. Twelve of the Cisco 2821 routers were 
installed between December 2008 and October 2010. The Office of Technology subsequently 
obtained the 55 Cisco model 2821 routers from the Tax Department via the surplus property 
process. OT installed the Cisco model 2821 routers in multiple State agencies with more users 
than most of the state libraries and State Police detachments that were given the Cisco model 
3945 routers. For example, OT placed one of the Cisco 2821 routers in the Martinsburg DMV 
office where it supports 22 users. 

Another example of a State agency which was not a part of the BTOP grant program, but 
purchased a Cisco router to support a new VoIP system is the Secretary of State's Office. The 
Secretary of State's Office in July 2011 purchased a Microsoft based VoIP system to service 75 
phone lines. The Secretary of State's Office purchased a Cisco model 2911 router to host their 
VoIP phone circuit. According to the Secretary of State's Chief Technology Officer, the Cisco 
291 1 router has adequately handled their VoIP phone circuit. 

Furthermore, the Legislature runs its network, not with a router but instead with a switch. 
The Legislature has a Cisco 3560 switch acting as our 'core' switch that handles upwards of 600 
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wired users for both data and VoIP telephone services. The Legislature has approximately 75 fax 
machines on that same network and upwards of 200 wireless users on any given day during 
session. The Legislature's switch is uplinked to a WVnet router, in a different building on the 
Capitol Complex, with 100 mbps fiber. Even at the busiest time of the legislative session with 
over 600 inside users and a large number of external users accessing multiple web servers, up to 8 
simultaneous live audio webcasts, multiple SQL servers, and multiple Google search appliances 
located in the Legislature's server farms, the network generally ranges around 40-50% bandwidth 
usage. It will burst above that during brief periods, but to the Legislature's network specialist 
knowledge, the Legislature never has exceeded the capacity of the switch or its connections to the 
WVnet router. When the Legislative Auditor asked one of the Legislature's network specialist 
whether he would like one of the Cisco model 3945 routers, he stated no because it greatly 
exceeds the Legislature's needs. 

Based on Cisco's own literature's recommendation for routers and the fact that multiple 
State agencies that have higher capacity functions than the state's public libraries and State Police 
detachments, have designed and operate networks with level 2 routers or with switches, including 
the Legislature, and the fact that schools with 500 or less students can meet the national 2017 
broadband access standards with level 2 routers, the Legislative Auditor concludes that the 
majority of Level 3 Cisco model 3945 routers which were purchased are unnecessary. 

The Legislative Auditor believes that the Cisco sales representatives and engineers had a 
moral responsibility to propose a plan which reasonably complied with Cisco's own engineering 
standards. It is the opinion of the Legislative Auditor that the Cisco representatives showed a 
wanton indifference to the interests of the public in recommending using $24 million of public 
funds to purchase 1,164 Cisco model 3945 branch routers. 
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Recommendations 



1. The State Purchasing Division should determine whether the actions or inactions 
by the Cisco representatives fall under the purview of §5A-3-33d of the West Virginia 
Code and are grounds for debarment. 

2. The State Office of Technology should immediately conduct a Capacity/Users 
Need study in conjunction with the State 's Broadband Deployment Council and report 
back to the Legislature prior to the conclusion of the 2013 Legislative Regular Session as 
to the results of such a study and whether it is legal to redeploy any of the Cisco model 
3945 routers to more appropriate public facilities to maximize their usage. Such a study 
should include determining whether the state 's smallest libraries even need level 2 router. 

3. The State Office of Technology should immediately contact Cisco and the NTIA to 
see if the State could trade out unnecessary features/modules in the Cisco model 3945 
routers which have yet to be deployed for the $80,000 of Cisco VOIP modules necessary 
for the Cisco model 3945s to run the State Police 's VOIP system. 
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Issue 2: A Legally Unauthorized Purchasing Process Was Used To Buy 1,164 
Cisco Model 3945 Routers At A Cost of $24 Million Using Federal Stimulus 
Funds 

The State Office of Technology used a purchasing process which is unauthorized by West 
Virginia statute or legislative rule to purchase 1,164 Cisco model 3945 branch routers at a cost of 
$24 million on behalf of the Broadband Technology Opportunity Program (BTOP) grant 
implementation team. The Office of Technology used a "Secondary Bid Process" on an existing 
contract approved by the State Purchasing Division, instead of a competitive bid process, open to 
non-Cisco vendors, as required by law. In addition to using the legally unauthorized secondary 
bid process, the contract used to purchase the routers was an inappropriate vehicle; the 2007 
IPT07D Contract with Verizon Network Integration CO was a statewide contract for a "simple 

12 

expansion" of telephony. However, the 1,164 Cisco routers purchased through the unauthorized 
process have the capabilities to support between 814,000 and 1.39 million voice over internet 
protocol (VoIP) telephones. 

I. Background 

After a request for proposal in 2007, the State of West Virginia entered into 
contracts with four companies in early 2008 "to establish a statewide contract for the 
procurement, installation, and on-going maintenance of Internet Protocol Voice (VoIP) 
communications equipment," and Cisco brand products or equal were specified because "the 
products we are requesting for purchase are a simple expansion of a pre-existing 
network/telephony hardware and software platform/system architecture." Alpha Technical 
Solutions, Alpha Technologies Inc., Pomeroy IT Solutions and Verizon Network Integration CO 
were awarded "statewide contracts" IPT07A, IPT07B, IPT07C and IPT07D, respectively. The 
contracts required that "any alternate products must seamlessly fit into, integrate with and 

12 See IPT07D Contract, p. 5. 

13 See, e.g., IPT07D Contract, p. 5. 
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interchange with the existing Cisco infrastructure investment with zero loss of feature 
functionality, and no infrastructure configuration changes." 14 

In 2010, 1,164 Cisco branch routers were purchased at a cost of $24,049,957.50 from 
Verizon (under the IPT07D Contract) using a purchase order (known as "ISCL0002"), which, 
according to the Purchasing Division, utilized a secondary bid process. 15 At issue is whether that 
purchase, and the secondary bid process itself, are legally authorized. In order to understand and 
evaluate the legal basis for the process used in the 2010 purchase, it is necessary to understand the 
statute and legislative rules that generally govern the purchasing process, as well as examine the 
Purchasing Division's Handbook and documentation concerning the purchase itself. 

II. Applicable Law 

a. West Virginia Code, Article 3, Chapter 5A 

Article 3, Chapter 5A of the West Virginia Code creates and provides for the procedures 
of the Purchasing Division (the "Division") within the Department of Administration. The 
Division exists "for the purpose of establishing centralized offices to provide purchasing, and 
travel services to the various state agencies." 16 Unless specifically exempted, the provisions 

17 

contained in Article 3 apply to all of the spending units of state government. The Division is led 
by a director, who has the power and duty to, among others, 

(2) Ensure that the purchase of or contract for commodities shall be based, 
whenever possible, on competitive bid; 



" Letter from David Tincher, CPPO, Director, Purchasing Division (January 17, 2013). 
16 W.Va. Code §5A-3-l(a). 

17 W. Va. Code §5A-3-l(c-d) (among others, the judicial and legislative branches are exempt from the provisions of 
this article). See also "Statutory Exemptions from the West Virginia Purchasing Division Approved by State 
Legislature," available at http://www.state.wv.us/admin/purchase/PurchasingExemptions/default.html (for the list 
maintained by the Purchasing Division). 
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[...] 



(9) Examine the provisions and terms of every contract entered into for 
and on behalf of the State of West Virginia that impose any obligation upon 
the state to pay any sums of money for commodities or services and 
approve each such contract as to such provisions and terms[. . .]; and 

(10) Assure that the specifications and commodity descriptions in all 
"requests for quotations" are prepared so as to permit all potential 
suppliers-vendors who can meet the requirements of the state an 
opportunity to bid and to assure that the specifications and descriptions do 
not favor a particular brand or vendor. ... 18 

The director is also required to propose rules for legislative approval to authorize a variety of 
purchasing methods, including direct and emergency purchases, the process for purchasing, 
including timelines, inspections, bonds, vendor registration, liquidated damages and cancellation 
provisions, contract management, oversight of purchases, surplus commodities. 19 Code of State 
Rules, Title 148, Series 1, Department of Administration, Purchasing Division contains the rules 
proposed by the director and approved by the legislature. Read together, the statute and rule 
provide and describe the authorized methods for purchasing, of which there are several. 

b. West Virginia Code of State Rules, Title 148, Series 1 

I. Methods of Purchasing 

1. Purchases of $25,000 or less 



W. Va. Code §5A-3-3(2),(9),(10). This is not a comprehensive list of the director's duties, but rather those most 
relevant to the purchasing process at issue here. See also W. Va. Code. R. §148-1-4 (describing the authority and 
remedies of the director). 
19 W. Va. Code §5A-3-4(a). 
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For purchases below $25,000, the director may make those purchases on the open market, 
but those purchases, wherever possible, must be based on at least three competitive bids. 20 
According to the legislative rule, "spending units may make purchases of $25,000 or less per 
transaction for certain commodities, services or printing pursuant to the most current Purchasing 
Division procedures and requirements established by the Director," so long as the spending units 
maintain records of those purchases for inspection. 21 The director also may authorize spending 
unit purchases without competitive bidding for $2,500 or less per transaction. The cost of 
maintenance and life of the commodity must be considered if the director determines there are 

22 

nationally accepted industry standards for the commodities. 

2. Purchases in Excess of $25,000 

Purchases estimated to exceed $25,000 must be made by the Purchasing Division, and the 
director must generally solicit sealed bids, by public notice as well as by request to prospective 
vendors. 23 

3. Open End Contracts 

'Open end contracts' are authorized by rule for "commodities, service or printing supplies 
to supply the repetitive needs of the State spending units in the form of statewide contracts, 

24 

blanket orders, or spending unit contracts." Agencies are required to use these contracts, unless 
granted an exemption by the director, and the director "may solicit requirements for similar 
commodities, services or printing to determine the best methods of acquisition." 



W. Va. Code. R. §148-1-7.2; W. Va. Code §5A-3-ll(a). 
W. Va. Code. R. §148-1-7.2. 
W. Va. Code §5A-3-ll(b). 

W. Va. Code §5A-3-10(b); W. Va. Code R. §148-1-6, 148-1-7.3. 
W. Va. Code. R. §148-1-7.4. 
W. Va. Code. R. §148-1-7.4. 
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4. Sole Source Procurement 



It is recognized throughout the Code that the competitive bid requirement either cannot be 
met in some instances, or need not be met. For example, the statute provides for situations in 
which competitive bidding might not be possible because there is only one source for the required 
commodity or service. 26 'Sole source procurement,' which means buying from one source 
without advertisement or competition, is appropriate when the director determines in writing that 
there is only one source, and the spending unit provides appropriate documentation regarding cost 
or pricing and the efforts made to determine availability of other sources. Prior to any such 
award, registered vendors will be informed and provided an opportunity to indicate an interest in 

97 

bidding on the commodity or service. The rules discourage the use of sole source purchases, 
giving the director power to reject requests by agencies "whenever competition is believed to be 
available," and encouraging agencies "to solicit competition rather than process a sole source 
request." 28 

5. Emergency Procurement 

Similarly, the director may authorize emergency purchases of specific commodities by a 
spending unit which exceed $25,000 upon written request, in the case of a bona fide emergency, 
such as delays by contractors, in transportation, or unanticipated work volume. Even in this 
situation, however, unless the requirement is waived by the director, "[competitive bids shall be 

90 

obtained if possible." 



lb W. Va. Code §5A-3-10c. 

27 W. Va. Code §5A-3-10c. 

28 W. Va. Code. R. §148-1-7.5.2 

29 W. Va. Code §5A-3-15; W. Va. Code. R. §148-1-7.6. 
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6. Requests for Proposals, Expressions of Interest & Best Value 
Procurement 

Another method authorized by rule, and which requires approval of the director, is the use 
of requests for proposal (RFP's) or expressions of interest (EOFs), which are awarded based on 

30 

criteria specified to bidders, including price (for RFP's). The director must determine in writing 
whether using 'best value procurement,' meaning awarding contracts based on criteria set forth in 
RFP's or EOFs, is in the best interest of the state, and awards must be made to the "highest score 
responsive and responsible bidder whose bid is determined, in writing, to be the most 

3 1 

advantageous to the state." Any such procurement must follow the same general process set 
forth for competitive bids in W. Va. Code §5A-3-10a. 

7. Purchases from Contracts Issued by Other Public Agencies and 
Entities (known as "Piggybacking") 

Fhe director is also authorized by rule to "approve a request by a spending unit to make a 
purchase from contracts issued by agencies of the federal government, agencies of other states, 
other public bodies or other state agencies," a process commonly known as 'piggybacking.' 
The director is also authorized to make cooperative purchases under this section, but in "all cases, 
these contracts or cooperative agreements shall be from valid properly awarded contracts and 
considered by the director to be available and financially advantageous and comparable to what 
can be obtained by competitive bid." Any spending unit wishing to piggyback must prove to the 
director that a) the contract would not conflict with any existing state contract, unless the prices 
are substantially lower than the state contract based on equal specifications, b) the contract will 
not cause a resident vendor to lose substantial business, unless the price difference is so great that 

30 W. Va. Code §5A-3-10b; W. Va. Core R. §7.7. 

31 W. Va. Code §5A-3-10b; W. Va. Core R. §7.7. 

32 W. Va. Code R §148-1-7.9; see also http://www. state. wv. us/admin/purchase/piggyback/default. html . 

33 W. Va. Code R §148-1-7.9.1. 
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it is in the state's best interest to piggyback, and c) that the contract would not cause extensive 
hardship to any spending unit offered preference under the code. The director must approve or 
disapprove any piggybacking. 34 

8. Multiple Awards 

The director is also authorized to award a contract to "one or more responsive and 
responsible bidders if the director determines in writing that a single award to an individual bidder 
would be insufficient: Provided, That the basis for the selection among multiple contracts at the 
time of purchase shall be the most practical and economical alternative and shall be in the best 
interests of the state." The written explanation of the necessity for multiple awards shall be 
based on several factors, and placed in the public contract file. 

9. Contract Management 

In addition to overseeing the purchasing process, the director's duties include prescribing 
"procedures by which oversight is provided to actively monitor spending unit purchases, 
including, but not limited to, all technology and software commodities and contractual services 
exceeding $1 million, approval of change orders and final acceptance by the spending units." 37 
The rules provide the contract management process, which the director may prescribe, for 
purchases of $1 million or less, as well as the mandatory contract management procedures for any 
purchase exceeding $1 million. The latter include post award conferences, monitoring, and status 
and activity reports to the director. 38 



W. Va. Code R §148-l-7.9.1(a-c); §148-1-7.9.2. 
W. Va. Code §5A-3-llc. 
W. Va. Code R §148-1-7.10. 
W. Va. Code §5a-3-4(a)(10). 
W. Va. Code R §148-1-7.13.2. 
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III. The Secondary Bid Process 



a. Purchasing Division Procedures Handbook 

According to the Director of Purchasing, the 1,164 Cisco branch routers from Verizon (the 
"IPT07D Contract") were purchased under a purchase order, which utilized a secondary bid 
process. The 'secondary bid process' is not specifically referenced or defined in either the Code 
or the Rules. The document entitled "Purchasing Division Procedures Handbook," currently 
defines the process as: 

A subsequent bidding process for commodities and services included in an 
existing contract that is required as part of the original terms and 
conditions. Participation in this secondary bid process is restricted to the 
pre-approved, certified vendors for the existing contract. 40 

The handbook further states that "some statewide or open-end contracts may require a secondary 
bid process," and refers agencies to the ordering procedures on statewide contracts for proper 
instructions. 41 

In addition to several Code and Rule sections, discussed below, the Director of Purchasing 
points to the Handbook's definition of the secondary bid process to explain a transaction that took 
place in 2010. 42 The term secondary bid process did not appear in the Handbook, however, until a 
revision on December 15 th , 2011. 43 The Handbook in effect at the time of the ISCL0002 
branch router purchase on July 15, 2010, contained no reference to the term "secondary bid 



Letter from David Tincher, CPPO, Director, Purchasing Division (January 17, 2013). 

Purchasing Division Procedures Handbook, 2.1 (Issued July 1, 2007, Revision 19 (November 27, 2012). 

Id., 4.7.2.1. 

Letter from David Tincher, CPPO, Director, Purchasing Division (January 2, 2013). 
Purchasing Division Procedures Handbook, Revision 15 (December 15 th , 2011). 
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process." Instead, the Handbook merely directed agencies to refer to ordering procedures on all 
statewide contracts for proper instructions. 45 

Further, the portion of the Handbook which eventually contained reference to the 
secondary bid process appears to fall under the section concerning Open-End Contracts, 
specifically, for 'releases' required to order commodities or services under the contract entitled: 
"Requiring Purchasing Division Approval." 46 In response to a request for information by the 
Legislative Auditor, the Director of Purchasing unequivocally stated as follows: 

As previously stated, the Purchasing Division was not required to review requirements or 
bids and/or approve purchase orders processed from the secondary bid process. 
Accordingly, the Purchasing Division had no knowledge of the $24 million router 
purchase order awarded through the secondary bid process. 41 

As noted above, the secondary bid process is not defined within the Code or the Rules. Further, 
the Handbook, even in its current form, does not contain any more detailed instructions about the 
secondary bid process, including who has the authority to approve the use of the process, what 
criteria should be used to judge the appropriateness or benefit to the state of using the process, 
and what type of purchases may be made using the process. According to the Director: 

[ajpprovals required to award a purchase order from the secondary bid process vary 
depending on the requirements contained in the master contract [. . .and] may include only 
the ordering agency, or they may include a combination of the ordering agency, the 
Purchasing Division, the West Virginia Attorney General's Office and any agency that is 
statutorily required to approve a purchase. 48 



Purchasing Division Procedures Handbook, Revision 9 (April 1, 2010). 
Id., at 4.7.1.2. 

Purchasing Division Procedures Handbook, Revision 15, at 4.7.1.2. 

Letter from David Tincher, CPPO, Director, Purchasing Division (January 29, 2013) (emphasis in original). 
Letter from David Tincher, CPPO, Director, Purchasing Division (January 2, 2013) (emphasis added). 
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This appears to differ from the Handbook, which implies that purchases using a secondary bid 
process require Purchasing Division Approval. It appears that once a 'master contract' is 
approved, the terms of that contract, as well as the vendors selected in that process, control the 
secondary bid process and any subsequent purchases, for as long as the contract remains in 
effect. 49 It is the opinion of the Legislative Auditor that the secondary bid process subverts the 
requirements of the purchasing statute and diminishes government oversight of procurement, thus 
jeopardizing the public trust and faith in good government. 

Finally, it should be noted that after repeated inquiries from the Legislative Auditor 
concerning the 2010 branch router purchase and the secondary bid process, the Director of 
Purchasing plans to again revise the Handbook to implement a new procedure for the secondary 
bid process. Specifically, effective February 1, 2013, 'releases' to order commodities or services 
from open-end contracts may be released directly by the agency if the purchases are anticipated to 
cost $250,000 or less, meaning those purchases may be made without advance Purchasing 
Division approval. When a secondary bid process is required, and the purchase may exceed 
$250,000, agency procurement officers must send a memorandum for original approval to the 
Purchasing Division including a synopsis of the purchase, the items, the possible participating 
vendors, and the anticipated amount. After the secondary bid process is conducted, the agency 
must submit similar evidence regarding the proposed award to the Purchasing Division for 
approval, after which point it may be processed. The statutory authority for this process, whether 
in its current form, or in the form proposed by the Director for implementation after February 1, 
2013 is unclear, for the reasons described below. 

b. The West Virginia Code and Code of State Rules 

The Purchasing Division points to W. Va. Code §5a-3-llc, which allows the award of 
multiple contracts. The code authorizes the director to award a contract to "one or more 

49 Purchasing Division Procedures Handbook, Revision 19, 4.7.2.1.; Letter from David Tincher, CPPO, Director, 
Purchasing Division (January 17, 2013); Letter from David Tincher, CPPO, Director, Purchasing Division (January 2, 
2013). 



38 lp g 



West Virginia Legislative Auditor Performance Evaluation & Research Division 



responsive and responsible bidders if the director determines in writing that a single award to 
an individual bidder would be insufficient: Provided, That the basis for the selection among 
multiple contracts at the time of purchase shall be the most practical and economical alternative 
and shall be in the best interests of the state." 50 The written explanation of the necessity for 
multiple awards shall be based on several factors, and placed in the public contract file. 51 Based 
on the documentation described as the BTOP router contract file, no such written explanation was 
placed in the file. Further, no documentation was provided indicating that any agency provided 
any written justification for a request to use multiple vendors. 52 Additionally, the code requires 
that any later selection among multiple vendors for purchase must be "the most practical and 

CO 

economical alternative and shall be in the best interests of the state." 

It appears that the Purchasing Division's interpretation of this statute would allow for use 
of 'the secondary bid process' for subsequent purchases under the contracts awarded to multiple 
vendors. This interpretation, however, assumes that the contracts awarded under the multiple 
award process are not subject to the other requirements of the code. First, the director points to W. 
Va. Code §5A-3-4(l) [sic] which according to his letter of January 2, 2013 "authorize[s] the 
director to . . . authorize a spending unit to purchase commodities directly and prescribe the 
manner in which such purchases shall be made." In fact, that section requires that the director 
"shall propose rules for legislative approval ... [to a]uthorize a spending unit to purchase 
specified commodities directly and prescribe the manner in which such purchases shall be 
made." 54 As noted above, those legislatively-authorized rules do provide for the direct purchase of 
commodities under certain circumstances, but do not appear to provide for a secondary bid 
process. Similarly, the director points to W. Va. Code §5A-3-4(3) [sic] and CSR 148-1-4.3, 
which "authorize the director to . . . prescribe the manner in which commodities shall be 
purchased." That subsection also refers to the director's duty to propose rules to prescribe the 
manner in which purchases should be made. Under the Purchasing Division's interpretation, this 

50 W. Va. Code §5A-3-llc. 

51 W. Va. Code R §148-1-7.10. 

52 W. Va. Code R §148-1-7.10.2. 

53 W. Va. Code §5A-3-llc. 

54 W. Va. Code §5A-3-4(a)(l). 
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subsection would authorize an otherwise non-legislatively-approved method of purchase (such as 
the secondary bid process), and would provide unlimited discretion to the purchasing director to 
make procurements in any way he wishes. It is the opinion of the Legislative Auditor that the 
totality of the code and the rules indicate that this was not the intent of the legislature. Rather, the 
purchasing director has the authority to propose rules for legislative approval prescribing the 
manner in which purchases shall be made, in accordance with the code. Those rules do not 
include the secondary bid process. 

Finally, the director points to W. Va. Code §5A-3-ll(e), which requires that "awards for 
open market orders, purchases based on advertised bid requests or contracts made by the director 
or a state department shall be awarded to the lowest responsible bidder or bidders." The fact that 
this requirement is included actually strengthens an alternative to the Purchasing Director's 
interpretation of the 'multiple award' section - that any award made under that section remains 
subject to the other requirements of the code - because if purchases based on multiple awards 
could be made using a secondary bid process, in essence exempting them from the other 
requirements of the code, this requirement would not apply. 

Based on a comprehensive analysis of the Code and the rules by the attorneys of 
Legislative Services, and based upon their advice, it is the opinion of the Legislative Auditor that 
the "multiple awards" section does not authorize a secondary bid process, but is rather a method 
by which the director may award multiple contracts, each of which is nonetheless subject to the 
other requirements of the Code for purchases of whatever type the original contract was intended 
to make. 

Further, it does not appear that the use of a secondary bid process is consistent with the 
Legislature's stated purpose of the purchasing statute: 

The Legislature of the State of West Virginia hereby declares that the purpose of 
this statute is to promote equal and fair bidding for the purchase of commodities 
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by the State and any political subdivision of the State purchasing commodities 
under any state contract; to eliminate fraud in the procurement of commodities by 
the state. 55 

It appears to the Legislative Auditor that in this instance the use of the secondary bid process took 
place wholly outside the control of the State Purchasing Division. The 2010 branch router 
purchase was inconsistent with the specific requirements of the purchasing statute. Had this 
purchase been competitively bid, the Purchasing Division would have administered the bidding 
process and provided contract management. In addition to exempting purchases from Purchasing 
Division scrutiny, the secondary bid process, as applied to the BTOP router purchase, appears to 
have the effect of limiting competition for future purchases to only those vendors on the original 
contract 



IV. The 2010 Purchase 



The State of West Virginia entered into contracts with four companies, including Verizon, 
in early 2008 "to establish a statewide contract for the procurement, installation, and on-going 
maintenance of Internet Protocol Voice (VoIP) communications equipment," and Cisco brand 
products or equal were specified because "the products we are requesting for purchase are a 
simple expansion of a pre-existing network/telephony hardware and software platform/system 
architecture." 56 The contracts required that "any alternate products must seamlessly fit into, 
integrate with and interchange with the existing Cisco infrastructure investment with zero loss of 

en 

feature functionality, and no infrastructure configuration changes." 



In 2010, the State purchased 1,164 Cisco routers from Verizon Network Integration Co., at 
a cost of $24,049,957.50 under the IPT07D Contract, using "Purchase Order ISCL0002." 58 This 
purchase was made under a 'secondary bid process,' which it appears was open only to the 

55 W. Va. Code §5A-3-30(a). 

56 See, e.g., IPT07D Contract, p. 5. 

57 Id. 

58 ISCL0002,p.l. 
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vendors originally awarded the IPT07 contracts. In addition to containing no reference to routers, 
the original IPT07 contracts specifically requested bids for "Cisco name brand product(s), "or 
equal"." 59 The release document for the 2010 purchase, however, stated that the "IPT07SWC 
[statewide contract] specifies Cisco equipment only." 60 Accordingly, it appears that, in addition 
to limiting the pool of bidders to those holding an IPT07 contract, only those bidders who could 
provide Cisco products were eligible to bid on the ISCL002 release. 

According to the Director of Purchasing, the Purchasing Division, a division within the 
Department of Administration, did not approve the decision to use the secondary bid process to 
make the 2010 router purchase under the IPT07D contract. The contract administrator for the 
IPT07D contract was the State Office of Technology, 61 also within the Department of 
Administration, which is the office responsible for evaluating and approving technology 
purchases. 62 In the case of the 2010 purchase, the statutory intent of several code requirements 
for a proper award was not met. First, the purchase exceeded $25,000, and thus should have been 
made by the Purchasing Division. It appears that only the Office of Technology approved the $24 
million purchase. Second, the process should have been based upon a more open competitive bid 
process. In 2007, the Purchasing Division conducted a prequalification process and prequalified 
four vendors to bid on "a statewide contract for procurement, installation, and on-going 

/TO 

maintenance of Internet Protocol Voice (VoIP) communications equipment." The responses did 
not contain estimated prices or a list of the specific equipment or services to be provided. In 
2010, only the four prequalified vendors were given notice and an opportunity to bid on the 
BTOP router purchase. Two of those vendors, Pomeroy and Verizon, submitted bids. It is not 
clear whether a determination was made that using a limited set of vendors met with the 
requirements of the code. This is likely because a secondary bid process was used, and that 
process does not appear to have any criteria for determining the appropriateness of its use. 



5y IPT07D Contract, p.5. 

60 ISCL0002, p. 14. 

61 Tincher Letter (January 17, 2013). 

62 W. Va. Code §5A-6-4, et seq. 
63 IPT07D, p. 5. 
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Finally, because the contract well exceeded $1 million, contract management procedures should 
have been followed. 

Conclusion 

The process used to purchase the Cisco routers was not equal, fair or consistent with the 
intent of the purchasing statute. The Legislature has declared that its intent is "to promote equal 
and fair bidding" and "to eliminate fraud." 64 Vendors of non-Cisco branch routers such as HP, 
Brocade, Juniper, and Alcatel-Lucent were not given notice or any opportunity to bid on the 
statewide expansion of broadband. Because the secondary bid process excluded all other branch 
router makers, the state cannot be certain that it received the best possible price for the routers, 
including the lowest possible price Cisco was willing to offer through its partner vendors. For 
example, in 2012, California State University, the largest four year university in America, used a 
competitive bidding purchase to purchase an eight year refreshing of its 23-campus 10G network. 
The Director of Cyber Infrastructure of California State University provided documentation 
showing that Alcatel-Lucent won the project with a bid of $22 million. Cisco's bid was $122.8 
million. The other bids were Brocade at $24 million, Juniper at $31.6 million, and HP at $41 
million. 65 Furthermore in May of 2011, Purdue University bid out replacement components for 
its Hansen Computer Cluster. 66 Cisco won the Purdue University competitive bid process by 
offering a 76% discount off the cost of its products. 67 

While the Legislative Auditor cannot say with certainty that a more open competitive bid 
process, required wherever possible by West Virginia law, would have resulted in a lower cost 
for the branch routers purchased by the BTOP grant, it is a basic principal of our free-market 



04 W. Va. Code §5A-3-30(a). 

65 California State Government documents. 

66 Hansen consists of Dell compute nodes with four 12-core AMD Opteron 6176 processors (48 cores per node), 
either 96 GB, 192 GB, or 5 12 GB of memory, and 250 GB of local disk for system software and local scratch storage. 
All nodes have 10 Gigabit Ethernet interconnects and a 5 -year warranty. 

67 Bradreese.com blog http ://www.bradreese.com/blog/8 -1-2011 .htm 



43lpg 



West Virginia Legislative Auditor Performance Evaluation & Research Division 



economy that competition between vendors leads to lower prices and higher quality products. 
Yet, instead of using an open competitive bid process to attempt to receive the lowest possible 
price and highest quality product, the State of West Virginia simply relied on Cisco's goodwill. 

Recommendations 

4. The Director of Purchasing should take action to immediately cease allowing any state 
purchases to be made through the legally unauthorized secondary bid process. 

5. If the Director of Purchasing believes that a secondary bid process should be allowed under 
West Virginia state law, the Director of Purchasing should request the Legislature to amend the 
purchasing statutes and legislative rules accordingly. 
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Appendix A: Objective, Scope and Methodology 

This Special Report on the 2010 Purchase of Routers With Broadband Technology 
Opportunity Program (BTOP) Funds is authorized by West Virginia Code §4-2-5. 

Objective 

The objective of this report was to examine the Executive Branch's implementation of the 
federal BTOP grant with regards to the purchase of computer routers. 

Scope 

This report examined the management of the BTOP grant with regards to computer routers 
from 2009 to February 2013. 

Methodology 

This review utilizes information from discussions and correspondence with multiple state 
agencies and individuals. These include: members of the original Grant Implementation Team 
still employed by the State, Mr. Jimmy Gianato, Director of Homeland Security and Emergency 
Management Division and Lt. Colonel Michael Todorovich; the Cisco engineer for the project, 
Mr. Mark Williamson; Mr. David Tincher, Director of the State Purchasing Division; the present 
State Chief Technology Officer, Ms. Gale Given; John Dunlap with the Office of Technology; the 
State Department of Education; the State Supreme Court of Appeals; the State Library 
Commission; and, the State's former Chief Technology Officer at the time of the purchase of the 
routers, Mr. Kyle Schafer. 
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